Fast Packet Forwarding on Commodity Platforms

Rather than using special-purpose hardware routers, software routers enable routing on commodity platforms. However, even with faster processors and multi-core platforms, the performance of software routers on commodity platforms today does not scale with high speed. We identify the limitations of commodity platforms by comparing them to high-end routers. In high end routers, each line card has its own memory for packet queues and the forwarding table (FIB), and its own processors to perform packet processing (e.g., forwarding table lookup). Packet processing is performed locally on line cards, and packets are forwarded to the outgoing interface through a dedicated high-speed switch. However, with a multi-core commodity platform, packets must be stored in a shared main memory and accessed by general-purpose processors. PCIe bus is used to transfer packets between main memory and line cards. As a result, we must leverage the hierarchical memory architecture and the processing resources in multicore commodity platforms. First, since packets are forwarded through a detour in the shared memory rather than directly between line cards, memory access becomes the main bottleneck, especially for small-size packets. (The PCIe bus becomes the bottleneck for largesize packets, which can be addressed with improved hardware techniques.) Therefore, software routers should leverage the small fast memory (cache) to improve their performance. Second, to best leverage computing resources with multiple cores, routing functions (e.g., packet forwarding, control plane) should be mapped to the appropriate place in the commodity hardware (e.g., different cores, NICs). Besides the limitations of commodity hardware, another reason for the poor performance of software routers is the different performance requirements between conventional applications and routing. In addition to throughput, routing is also sensitive to delay and delay jitter. Moreover, routing requires more predictable behavior under a range of workloads than conventional applications, especially under worst-case workloads (e.g., a burst of packets with a wide range of destinations, or routing changes due to network events). Based on the limitations of commodity hardware and the specific performance requirements of routing, we propose a fast packet forwarding mechanism leveraging today’s multi-core commodity platforms, which works well under worst-case workloads. We also propose enhancements for future multi-core commodity hardware to further improve the performance of software routers.